Research Questions

Consider this question: “Does Trump’s anti-vaccine rhetoric reduce vaccination rates among his supporters?”

Three possible approaches:

• Approach 1: Survey people about Trump and their vaccination status
• Approach 2: Compare vaccination rates in Trump vs. Biden counties
• Approach 3: Track the same people’s vaccination status before and after seeing or not seeing Trump statements on vaccines

Research Design Taxonomy

graph TB
    A["🔬 Research Designs<br/>for Political Science"]
    
    A --> B["🎯 Experimental<br/>Designs"]
    A --> C["🔍 Observational<br/>Causal Designs"]
    A --> D["📊 Observational<br/>Descriptive Designs"]
    
    B --> E["🎲 Survey<br/>Experiments"]
    B --> F["⚖Lab<br/>Experiments"]
    
    C --> H["🌿 Natural<br/>Experiments"]
    C --> I["📈 Regression<br/>Discontinuity"]
    C --> J["⏱️ Difference-in-<br/>Differences"]
    
    D --> K["📋 Cross-sectional<br/>Surveys"]
    D --> L["📚 Case<br/>Studies"]
    D --> M["🔄 Panel<br/>Studies"]
    
    classDef startNode fill:#1a2332,stroke:#4fc3f7,stroke-width:3px,color:#e3f2fd
    classDef easyNode fill:#1b2d1b,stroke:#4caf50,stroke-width:2px,color:#e8f5e8
    classDef endNode fill:#1a2332,stroke:#00bcd4,stroke-width:3px,color:#e3f2fd
    classDef difficultNode fill:#2d1b17,stroke:#ff9800,stroke-width:2px,color:#fff3e0
    
    class A startNode
    class B,E,F,G easyNode
    class C,H,I,J endNode
    class D,K,L,M difficultNode

What Makes an Experiment?

Two key features:

1. Random Assignment: Researchers control who gets what treatment
2. Control Groups: Direct comparison between treated and untreated groups

Why Random Assignment Matters

The Problem: People are different in many ways

• Some people always vote, others never do
• Some are more politically engaged
• Some have more flexible schedules

Why Random Assignment

Without random assignment, groups can be very different:

What is Experimental Control?

• Control group: A comparison group that doesn’t receive the treatment

Without control, we cannot make causal claims

• Example: If we only study people who received voter mobilization calls, we can’t know if their high turnout was due to the calls or because they were already motivated voters (on the voter rolls)

What Are We Controlling For?

The threats to valid causal inference

How Random Assignment Works

Random assignment: Each subject has an equal chance of being assigned to any group

flowchart TD
    subgraph Initial["Initial Population"]
        B1["🔵 Blue"] 
        B2["🔵 Blue"]
        R1["🔴 Red"]
        B3["🔵 Blue"] 
        R2["🔴 Red"]
        R3["🔴 Red"]
        R4["🔴 Red"]
        B4["🔵 Blue"]
    end
    
    Initial --> Coin["🪙 Coin Flip<br/>Each Person:<br/>Heads = Group 1<br/>Tails = Group 2"]
    
    Coin --> Group1["Group 1 (Treatment)<br/>🔵🔵🔴🔴"]
    Coin --> Group2["Group 2 (Control)<br/>🔵🔵🔴🔴"]
    
    classDef startNode fill:#1a2332,stroke:#4fc3f7,color:#e3f2fd
    classDef coinNode fill:#2d1b17,stroke:#ff9800,color:#fff3e0
    classDef endNode fill:#1b2d1b,stroke:#4caf50,color:#e8f5e8
    
    class Initial startNode
    class Coin coinNode
    class Group1,Group2 endNode

Random Assignment: Unobserved Characteristics

In practice, we often don’t know people’s characteristics beforehand

flowchart TD
    subgraph Initial["Population (8 People)"]
        U1["❓ Unknown"] 
        U2["❓ Unknown"]
        U3["❓ Unknown"] 
        U4["❓ Unknown"]
        U5["❓ Unknown"]
        U6["❓ Unknown"]
        U7["❓ Unknown"]
        U8["❓ Unknown"]
    end
    
    Initial --> Coin["🪙 Coin Flip<br/>Each Person:<br/>Heads = Group 1<br/>Tails = Group 2"]
    
    Coin --> Group1["Group 1 (Treatment)<br/>🟢🟢🟠🟠<br/>2 Green + 2 Orange"]
    Coin --> Group2["Group 2 (Control)<br/>🟢🟢🟠🟠<br/>2 Green + 2 Orange"]
    
    classDef startNode fill:#1a2332,stroke:#4fc3f7,color:#e3f2fd
    classDef coinNode fill:#2d1b17,stroke:#ff9800,color:#fff3e0
    classDef endNode fill:#1b2d1b,stroke:#4caf50,color:#e8f5e8
    
    class Initial startNode
    class Coin coinNode
    class Group1,Group2 endNode

Random Assignment: Individual Trial Variation

Random assignment doesn’t guarantee perfectly balanced characteristics in any single trial (100 people, 50 purple/50 pink, randomly assigned to groups of 50 each)

The Law of Large Numbers

What happens when we run many random assignments? (100 people, 50 purple/50 pink, groups of 50 each)

Some Political Science Experiments

Political science has embraced experimental methods in recent decades

Why experiments matter in political science:

• Causal claims: Experiments allow us to make definitive statements about cause and effect
• Policy relevance: Results directly inform campaign strategies, government programs, and civic interventions
• Theory testing: Experiments can definitively test competing theoretical predictions
• Real-world impact: Experimental findings shape how campaigns run, how governments communicate, and how civic organizations operate

Experiment 1: Get Out The Vote

Gerber & Green (2000) - Do campaign methods actually work?

Results: Personal canvassing increased turnout by 7-10 percentage points; phone calls and mail had minimal effects

Experiment 2: Media and Reconciliation in Rwanda

Paluck (2009) - Can media promote peace after conflict?

Results: Educational radio soap opera significantly changed social norms and behaviors toward reconciliation

Experiment 3: Deep Canvassing and Transgender Rights

Broockman & Kalla (2016) - Can prejudice be changed through conversation?

Results: 10-minute conversations reduced transgender prejudice for at least 3 months

When You Can’t Randomly Assign

Some research questions involve things you can’t control:

• Constitutional changes (can’t randomly assign constitutions to countries)
• Major life events (can’t randomly assign parents to divorce)
• Policy implementations (can’t randomly assign laws to states)

Solution: Find situations where assignment was “as good as random”

What Does “As Good As Random” Mean?

When nature or institutions create randomization for us

Examples of “as good as random” assignment:

• Draft lottery: Birthdays randomly determine military service
• Policy rollout: Budget constraints mean only some areas get new program first
• Natural disasters: Random timing affects some regions but not others
  
• Administrative cutoffs: Arbitrary thresholds (age 65 for Medicare) create treatment groups

Natural Experiments

Three examples of “as good as random” assignment in the real world

Natural Experiment 1: The Draft Lottery

Angrist (1990) - Does military service affect lifetime earnings?

Results: Military service reduced lifetime earnings by about 15% due to lost work experience

Natural Experiment 2: Rainfall and Economic Voting

Achen & Bartels (2016) - Do voters blame politicians for things beyond their control?

Results: Voters punish incumbent politicians for poor economic conditions caused by weather, not policy

Natural Experiment 3: Female Political Representation

Chattopadhyay & Duflo (2004) - Do female leaders prioritize different policies?

Results: Villages with female leaders invested significantly more in drinking water infrastructure and roads

Once? Multiple times?

How often can we collect data: Once (cross-sectional) or multiple times (repeated cross-sectional or longitudinal)?

Cross-Sectional Data: Snapshots in Time

Longitudinal Panel Data: Following the Same People

The World’s First Data Scientist/ founder of demography

What Graunt Discovered

Before Graunt: People thought plague deaths were random acts of God

Simulation of Graunt’s Analysis:

# A tibble: 4 × 4
  season avg_plague_deaths avg_total_deaths total_parishes
  <fct>              <dbl>            <dbl>          <int>
1 Winter               2.1              7.4             20
2 Spring               3.3             12.4             20
3 Summer               5.3             19.7             20
4 Fall                 3               11               20

1. Patterns exist: Death rates weren’t random - they followed predictable patterns
2. Seasonal variation: Clear seasonal patterns with highest deaths in summer (plague season), lowest in winter
3. Demographic regularities: Consistent ratios of male to female births across time
4. Population estimation: First systematic attempt to count London’s population using data

Example: Voter Turnout Research

Question 1: “What percentage of Americans vote in midterm elections?”

• Best Design: Descriptive survey or administrative data analysis
• Why: Simple descriptive question requiring representative data

Question 2: “Do voter reminders increase turnout?”

• Best Design: Randomized field experiment
• Why: Can randomly assign reminders, ethical to do so, clear causal question

Question 3: “Does early voting increase overall turnout?”

• Best Design: Natural experiment or difference-in-differences
• Why: Can’t randomly assign electoral rules, but can compare before/after adoption

Internal vs. External Validity

Internal Validity: Are your results correct for your specific study?

• Experiments: High internal validity (random assignment eliminates confounding)
• Observational studies: Lower internal validity (many potential confounders)

External Validity: Do your results apply to other people, places, times?

• Experiments: Often lower external validity (artificial settings, selected populations)
• Observational studies: Often higher external validity (real-world settings)

The Fundamental Tradeoff

Your choice depends on your research question and what tradeoffs you’re willing to accept

Key Concepts to Remember